Online Apprenticeship Learning

In Apprenticeship Learning (AL), we are given a Markov Decision Process (MDP) without access to the cost function. Instead, observe trajectories sampled by an expert that acts according some policy. The goal is find policy matches expert's performance on predefined set of functions. We introduce online variant AL (Online Learning; OAL), where agent expected perform comparably while interacting with environment. show OAL problem can be effectively solved combining two mirror descent based no-regret algorithms: one for optimization and another learning worst case cost. By employing optimistic exploration, derive convergent algorithm O(sqrt(K)) regret, K number interactions MDP, additional linear error term depends amount available. Importantly, our avoids need solve MDP at each iteration, making it more practical compared prior methods. Finally, implement deep which shares similarities GAIL, but discriminator replaced costs learned OAL. Our simulations suggest performs well in high dimensional control problems.